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[APPARATUS FOR MATCHING X-RAY IMAGES WITH REFERENCE IMAGES/ 

BACKGROUND OF THE INVENTION 

Field of the Invention 

This invention relates to matching similar x-ray images and has 
particular application to computer controlled radiotherapy apparatus for automatically 
matching on-line the portal images generated during radiotherapy treatment on a 
5 treatment machine with simulation images generated prior to treatment on a simulation 
machine for determining that the desired target is actually being irradiated for the 
purposes of assessment, and/or controlling the treatment equipment. 
Background Information 

There are medical applications which require matching of x-ray images. 

10 For instance, in computer controlled radiotherapy, treatment beams of high energy 
radiation are directed at a tumor from a number of directions so as to maximize 
irradiation of the tumor while minimizing exposure of healthy tissue surrounding the 
tumor. Such radiotherapy treatment typically has two distinct phases: the simulation 
phase, and the actual treatment phase. In the simulation phase, the patient is placed 

15 on equipment similar to the treatment equipment except that it does not generate the 
high energy radiation beam. The simulation equipment is successively positioned to 
simulate the delivery of the sequence of treatment beams prescribed by the treating 
oncologist. This assures that the equipment can be positioned to deliver the required 
treatment beams and progressively move from one treatment beam to the next without 

20 collision between the equipment and the patient or between movable components of the 
equipment. During this procedure a low dosage x-ray image called the simulation 
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image is taken. This simulation image, which generally has good contrast and detail 
because of the low energy of the x-ray beam used (in the kiloelectronvolt range) helps 
the oncologist to locate the position of the tumor and thereby establish the positions of 
the equipment components for delivering the successive treatment beams. 

During the actual treatment phase, the patient is placed in the exact same 
position on the equipment as in the simulation before the regular-dosage x-ray 
radiation, typically in the megaelectronvolt range, is used to treat the patient. During 
this phase, another x-ray image is taken, which is called the portal image. 

After completion of the treatment, the simulation and portal images are 
compared by an expert to determine whether the tumor, as identified in the simulation 
image, has been adequately treated with radiation in the portal image. If the coverage 
is not complete, the patient is scheduled for a corrective treatment. 

The current accepted procedure involves the manual comparison of the 
portal and simulation images. Accurate manual comparison is quite challenging given 
the fact that the two x-rays are usually taken by different equipment and at different 
levels of radiation exposure. The latter fact implies that the tumor area is usually not 
visible in the portal x-ray, and thus the matching of the portal image with that of the 
simulation has to rely on manual estimation of dimensions from anatomical landmarks, 
which will not be clearly visible. 

Conventionally, the portal images have been generated by using x-ray 
film which has to be developed. This is not a serious drawback where only a single 
or a few treatment beams are utilized. However, this x-ray film is a serious limitation 
in computer controlled radiotherapy where a large number of treatment beams are used. 
Electronic portal imagers have been developed which generate a digitized image which 
can be displayed on an electronic display device. Unfortunately, the same problems 
exist as to the contrast and definition in the portal image generated electronically. 

The problem of matching portal images with simulation images is 
compounded by the fact that the images have differences in orientation caused by 
skewing, scaling differences, rotation, translation and differences in perspective and 
curvature. 

In stereotactic radiology, digitized computed tomography x-ray images 
and magnetic resonance images (MRI) have been automatically matched by applying 
scaling derived from known fixed dimensions of a steel frame which appears in both 
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images. Such fixed landmarks of known dimensions are not available in conventional 
radiotherapy images. 

There is a need, therefore, for apparatus for automatically matching x- 
ray images and particularly for matching portal images with simulation images in 
radiotherapy. 

There is also a need for such apparatus which can match the portal and 
simulation images on-line for multiple treatment beams. 

There is further need for such apparatus which can match portal images 
and simulation images having widely different contrast and definition and differences 
caused by skewing, rotation, scaling, perspective or curvature. 

There is an additional need for apparatus for obtaining and maintaining 
alignment of a patient during computed controlled radiotherapy or for terminating the 
radiation beam if alignment becomes unacceptable. 

SUMMARY OF THE INVENTION 

These needs and others are satisfied by the invention which is directed 
to apparatus for automatically matching an x-ray image with a reference image, and 
particularly for matching the portal image with a simulation image for determining 
whether radiotherapy treatment has been adequate or for matching successive portal 
images for controlling operation of the radiotherapy equipment. In matching images, 
digitizing means digitizes the x-ray image such as the portal image to generate a first 
set of digital image signals or digital portal image signals (DPIS) in the case of the 
portal image. The digitizing means also digitizes the reference image such as the 
simulation image to generate second digital image signals or digital simulation signals 
(DSIS). Processing means process these digital image signals to generate matched 
digital image signals. The processing is performed without any prior knowledge of the 
physical dimensions of any of the features in the images. Output means generate for 
instance a display from the matched digital image signals and/or control the 
treatment/diagnosis equipment. 

The processing means includes coarse alignment means which first effect 
a coarse alignment between the digital portal image signals and the digital simulation 
image signals. Coarse alignment is initiated by selecting seed points in the portal 
image represented by the DPIS and in the simulation image represented by the DSIS. 
Selection of the seed points can be done either interactively using a pointing device 
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such as a mouse to select what appear to be corresponding points on displays of the 
two images, or automatically through use of x-ray opaque fiducials placed on the 
patient. In either case, the seed points are used to compute a transform between the 
two images. Means are then used to apply the transform to one of the sets of digital 
5 image signals to transform points in that image to the coordinates of the other image 
thereby producing coarse aligned DPIS and DSIS. 

Following coarse alignment, a fine alignment is performed. In 
implementing the fine alignment, the coarse aligned DPIS and DSIS are first prepared 
by selecting selected DPIS and selected DSIS for regions of the images which intersect 

10 or overlap, and preferably for a region of regular shape such as a rectangle within the 
intersecting regions of the images. The digital image signals for these regions are then 
enhanced to produce prepared images with similar dynamic range and pixel intensities. 
The fine alignment means includes means generating an updated transform from the 
prepared DPIS and DSIS, and means applying the updated transform to either the 

15 coarse or prepared DPIS and DSIS to generate the matched DPIS and DSIS. 

The means generating the updated transform comprises means generating 
motion flow components from the prepared DPIS and DSIS and calculating means 
calculating the updated transform from the motion flow components. Preferably the 
means generating the motion flow components generates motion flow gradient 

20 components and the calculating means comprises means applying a robust optimization 
to calculate the updated transform. The means generating updated transform uses 
successive ascending levels of resolution of the prepared DPIS and DSIS to generate 
the updated transform. 

In the tracking mode, the updated transform is used to track movement 

25 between successive sets of digital portal image signals. Tracking can be used to 
terminate the radiation if patient movement exceeds specified limits, or could be used 
to operate the patient positioning assembly to maintain the radiation beam in proper 
alignment with the area to be treated. 

The invention can also be used to match x-ray images with other 

30 reference images which could be another x-ray image or another type of image. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
A full understanding of the invention can be gained from the following 
description of the preferred embodiments when read in conjunction with the 
accompanying drawings in which: 
5 Figure 1 is a schematic diagram of apparatus for implementing the 

invention. 

Figure^a is a simplified illustration of a simulation image to which the 
invention can be^pplied. 

y^Figure 2b is a simplified illustration of a portal image to which the 
10 inventioiymay be applied. 

/ Figure 2c is a simplified illustration of a display superimposing the 
simulation and portal images of Figures 2a and 2b utilizing the invention. 
/ Figures 3-11 are flow charts of software utilized to implement the 

invention in the apparatus of Figure 1 . 
15 DESCRIPTION OF THE PREFERRED EMBODIMENT 

The invention is directed to matching x-ray images with reference images 
and will be described as applied to matching portal images generated in computer 
controlled radiotherapy with simulation images. However, it will be understood that 
the invention has wide application in matching other x-ray images such as those used 
20 in diagnosis, for example. As will be seen, the invention also has application for 
tracking motion in successive portal images such as for controlling positioning of a 
patient or gating of the radiation beam. 

Referring to Figure 1 , a simulation setup 1 is used for determining the 
location of the region such as a tumor within a patient 3 to be treated and for setting 
25 up the sequence of treatment beams. The setup equipment includes a gantry 5 mounted 
for rotation about a horizontal pivot 7 supported by a machine base 9. A low energy, 
in the kiloelectronvolt range, x-ray beam 1 1 is directed by a collimator 13 mounted on 
the gantry 5 along a path which extends transversely through an extension of the pivot 
7. 

30 The patient 3 is supported on a patient positioning assembly 15 which 

includes a couch 17 mounted on a support 19 for three dimensional translation relative 
to the support. The support 19, in turn, is mounted on a turntable 21. Through 
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translation of the couch 17, rotation of the turntable 21 and rotation of the gantry 5 
about the pivot 7, a plurality of treatment beams can be simulated. By sequencing the 
simulation equipment 1 through the positions required to generate the successive 
beams, it can be determined whether all of the required beams can be achieved and 
5 whether sequencing the movement of the equipment between beams must be adjusted 
to avoid collisions between the equipment and the patient or between components of 
the equipment. 

The low energy x-ray beam 1 1 is used to generate simulation images by 
placement of an x-ray film 23 in line with the x-ray beam 1 1 on the other side of the 

10 patient 3 from the collimator 13. This simulation image is used to position the area 
of the patient to be treated, such as a tumor, at the isocenter of the setup, which is the 
intersection of the beam 11 with a projection of the pivot axis 7. 

Following completion of the simulation, the patient 3 is transferred to 
the treatment setup 1'. As shown, the treatment setup at V is similar to the simulation 

15 setup 1, except that the x-ray beam 11' is in the megaelectronvolt range. A portal 
image is generated by the treatment setup 1 ' . This portal image can be captured by an 
x-ray film as in the simulation setup; however, it is preferred that an electronic portal 
imager 25 be used. If available, an electronic imager could also be used in place of 
the x-ray film 23 in the simulation setup 1 . 

20 As discussed above, the simulation image and the portal image can be 

quite different. One of the main reasons for this is the difference in the energy of the 
beams 11 and 11' . The invention can be used to match the simulation and portal 
images to determine if the treatment dosage was delivered to the proper treatment area. 
It can also be used to detect patient movement during treatment to terminate generation 

25 of the x-ray beam 11' if a movement exceeds proper limits, or to maneuver the 
equipment to maintain proper alignment. 

The image matching system 27 includes a digitizer 29 which digitizes 
the simulation image such as produced on the x-ray film 23 and the portal image such 
as that generated by the electronic portal imager 25. In a more general sense, the 

30 matching system 27 matches an x-ray image, such as the portal image, with a reference 
image such as the simulation image. 

The image matching system 27 further includes a processor 31 which 
includes a module for coarse alignment 33 followed by a module for fine alignment 35. 
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The output of the processor can be matched portal (x-ray) and simulation (reference) 
images which are displayed on a display device 37. Associated with the display device 
37 are interface devices 39 which can include a keyboard 41 and a pointing device 43, 
such as a mouse or a trackball. 
5 Figures 2a-2c illustrate that the invention can be used to match a portal 

x-ray image with a simulation reference image. Figure 2a represents a simulation 
image 45 generated using the simulation setup 1. The low energy x-rays used for this 
image produce an image with good contrast and detail, so that the outline 47 of the 
patient and bony structure 49 are shown as well as the tumor 5 1 . Figure 2b illustrates 

10 the portal image which being taken with the higher energy treatment beam shows the 
treated area 55 as a uniform dark spot. The irregular edge of the treated area 55 is 
produced by the leaves used in the collimator 13 to conform the beam 11' generally 
to the shape of the tumor. The remainder of the portal image 55 shows little detail and 
does not indicate the location of the bones. As can be seen, the two images 45 and 53 

15 can be translated relative to each other, scaled differently, skewed and rotated (by 90° 
in the example). The two images can also be different in perspective and in curvature. 

The coarse alignment module 33 produces a general alignment of the two 
images, and then the fine alignment module 35 uses robust motion flow to rapidly and 
accurately complete matching of the images. The display device 37 can present the 

20 matched images in different ways. In one embodiment, the display 37 alternates 
between the two images at about 6 to 20 Hz, but usually about 12 Hz, so that the 
observer views the images superimposed as a composite image 59, as shown in Figure 
2c. As can be seen in the example, the treated area 55' in the matched portal image, 
overlays the tumor 51' in the matched simulation image. In another type of display 

25 (not shown), the outline of the treated area from the portal image is projected onto the 
processed simulation image, so that it can be seen if the targeted tumor was in fact 
treated. 

In performing the coarse alignment, a coarse transformation is applied 
to the digitized x-ray or portal image signals (DPIS) to convert them to the coordinate 
30 system of the digital reference or simulation image signals (DSIS). As will be seen, 
the information needed to generate this transformation can be generated interactively 
through selection of what appear to be corresponding points in the two images by the 
operator interactively using a pointer device 43 or automatically using x-ray opaque 





- 8 - 127442 

fiducials 61 which are placed on the patient in both the simulation setup and the 
treatment setup (see Figure 1). The points so generated in either case are referred to 
as seed points. The coarse transform H from the portal image coordinates to the 
simulation coordinates is: 



simulation x 
simulation 



RotSkewScale a RotSkewScale b translation x 
RotSkewScale c RotSkewScale d translation y 





portal x 




portal y 




1 



(EQ. 1) 



The (x y) vector denotes the column and row coordinates of the center 
of each of the seed points in the corresponding portal and simulation images. The four 
RotSkewScale components of the matrix describe the full affine transformation that is 
needed to coarsely align the images. In this stage, the placement of the fiducial or the 
interactive selection of the seed points need not be accurate as the next stage is able to 
accommodate for reasonably small deviations. 

Using the results of the coarse alignment, the portal image is warped 
toward the simulation image. Then, overlapping regions of the two images are 
computer enhanced so that the corresponding intensity levels are similar. Finally, the 
motion-flow, or the fine-scale transform is computed so that the portal image glides on 
the gradient of dissimilarity toward the simulation image. In this stage, a more 
comprehensive transformation model is used in which the input position vector is 
represented by: 



X(x) = 



1 x y 0 0 0 x 2 
0 00 1xyx-y 



y o 



(EQ.2) 



and the transformation matrix is represented by: 



Q = [ a 0 «1 a 2 tt 3 a 4 a 5 PO Pi C Y (EQ ' 3) 



so that the result is: 
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u (x;Q)=X (jc) • Q (EQ. 4) 

whereA portal (x;Q) = u(x;Q) and portal (jc) = X (jc) . The parameters a 0 through a 5 

include the affine transform as in the coarse alignment, whereas the parameters/^, p x 

include the perspective transformation, and c covers the deformation that can be caused 
by breathing, etc. 

To recover the parameters of the vector Q we formulate the image 
dissimilarity as a result of motion-flow, or distance between the two images. 

I(x,t) = I{x - (X(x) ■ <? />f+1 )) (EQ. 5) 



for Vjc e /, where /is the region of the image we compute the transformation over. 
In (EQ. 5), l(x) is the intensity function at point x> the image at / + 1 is the portal 
image, and at t is the simulation image. By using various derivation techniques, we 
formulate the motion-flow using the gradient (or dissimilarity gradient) as below: 



dt 



V/aOt) * Q f ) + — = 0 (EQ. 6) 



for Vx e f. 

In this stage, a robust regression method is employed, using 
unconstrained optimization, to calculate the elements of Q (see (EQ. 3)). This 
technique enables us to cope with the 'reasonably small' deviations from the coarse 
alignment stage, as well as any residual dissimilarity between the two images. Using 
the robust technique ensures that only the dominant transformation will be recovered 
without running into the risk of being affected by the noise and residual errors. 

Figures 3-11 are flow charts of software which implements the 
invention. Figure 3 illustrates the main routine 100 which includes performing a 
coarse alignment, either interactively at block 1 10 or automatically at block 120. In 
both cases a rough approximation of the transformation between the portal image and 



O 
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the simulation image is calculated using Equation 1 . The user then has the option of 
determining whether the rough approximation has provided a satisfactory alignment of 
the images at 130. If so, the procedure is completed. If not, a fine alignment is 
performed. As discussed, the invention can also be used to track patient movement, 
in which case the transformation between the two images is utilized at 140 to roughly 
determine the updated position of the fiducials. If requested by the user in image 
matching and during tracking, the images are prepared for the fine alignment at 150. 
The refined image transformation is then calculated at 160 and if the image matching 
mode is selected as determined at 170, the transform is accomplished and the images 
are displayed at 180 in the manner discussed above. If the tracking mode has been 
selected at 190, the routine returns to 140 for generating the next position. The user 
again has the final decision at 200 to determine whether the image matching is 
satisfactory. If not, the routine returns to 1 10 and the rough calculation is re-initiated. 

The procedure for calculating the rough approximation of the 
transformation interactively called for at block 110 in Figure 3 is illustrated in detail 
in Figure 4. The user selects corresponding seed points or areas in the portal image 
and the simulation image using, for instance, the mouse 43 as indicated at 111. The 
selected areas or points are then used to compute the rough transformation between the 
portal image and the simulation image by calling a procedure A as indicated at 112. 
This rough transform is then used to transform the portal image to simulation image 
coordinates by calling procedure B as indicated at 113. The images are then displayed 
on the monitor 37 as indicated at 114. 

The details of procedure A used to calculate the rough transform are 
shown in Figure 5. If the user has indicated an area as determined at Al, the system 
automatically selects random points from inside the area as corresponding as indicated 
at A2. Then, or if the user has selected points rather than an area, the corresponding 
point pairs are used to calculate the transform parameters using the least squares (LSQ) 
method as indicated at A3. 

The details of procedure B for transforming the portal to simulation 
coordinates is shown in Figure 5. First, the row and column limits of the resulting 
transformed portal image are determined at Bl using the transformation matrix H, 
which is the inverse of Equation 1 . The resulting portal image is then raster scanned 
at B2, and for each pixel the location is determined using the transformation. The 
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intensity value for each pixel is calculated next using linear interpolation between the 
surrounding pixel locations in the original porte^-ifriage. 

The routine 124 for performitig the coarse alignment automatically using 
fiducials on the patient is shown^irf^Figure 7. The x-ray opaque fiducials 61 are 
5 detected in both the portal ^and simulation images at 121 and the corresponding markers 
are identified at 122. "^fhe image transform is then computed at 123 using procedure 
A of Figu^5"and the centroid of each of the markers as the seed points. The portal 
image is^then transformed to simulation coordinates using the computed transformation 
^ancfprocedure B of Figure 6. When in the matching mode as determined at 125, the 
10 images are displayed at 126 in the manner discussed above in connection with Figures 
2a-c. 

The routine 150 for preparing the coarse aligned digital image signals 
for fine alignment is shown in Figure 8. First, the region of intersection over overlap 
between the simulation and portal images is calculated at 151 using the transformation 

15 of Equation 1. Next, the largest rectangular region that fits within the intersection 
region is calculated at 152. Other regular geometric shapes, such as a square and so 
forth, could be used in place of the rectangle. New images representing the 
rectangular intersection region of the portal and simulation image are formed at 153. 
These resulting images are then enhanced at 154 to generate prepared digital image 

20 signals. Various forms of enhancement such as histogram equalization, lapalcian of 
the Gaussian, high-pass filtering and other techniques are used to produce the prepared 
images with similar dynamic range and pixel intensities. 

Figure 9 illustrates the routine 160 for calculating the updated 
transformation for a fine alignment. This process is performed at several levels of 

25 resolution of the digital image signals beginning with the lowest resolution, which in 
the example is about one-eighth resolution. Thus, at 161 the images at the lowest 
resolution for the prepared portal and simulation images are formed. These images are 
updated using the latest updated transformation parameters, that is, transformation 
parameters calculated at the previous level of resolution, at 162. An important part of 

30 the invention is that robust motion flow is used to perform the fine alignment. In 
particular, the motion flow gradient components are generated at 163. Application of 
motion flow using gradient components is described by M. J. Black and P. Anandan 
in a paper entitled, n A Framework For The Robust Estimation Of Optical Flow" 
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published in Proc. 4th Intl. Conf. on Computer Vision (ICCV 93), Berlin, Germany, 
May 1993. Motion flow is applied to the motion required to cause pixels on one image 
to flow into alignment with corresponding pixels in the other image. Robust motion 
applies to the motion by which most of the pixels which have moved have moved 
5 similarly, while there may be others exhibiting different motion. The updated image 
transformation parameters are then calculated at 164 using robust optimization. If the 
upper limit of resolution has not been reached as determined at 165, then the resolution 
is incremented at 166 and updated transformation parameters are recalculated at the 
new level of resolution. 

10 When the highest level of resolution has been reached at 165, the final 

transformation matrix Q is generated at 167. The details of the routine for calculating 
the updated image transformation parameters using robust optimization of block 164 
in Figure 9 is shown in Figure 10. As described in the paper by Black and Anandan 
discussed above, the robust motion is represented by data points called inliers. Those 

15 exhibiting other motion are identified as outliers. In the present invention, the data 
points are the pixel values. The pixels are successively separated into inliers and 
outliers based upon their contribution to a consistent motion flow vector. The pixels 
in the inlier set are used to calculate the dominant motion flow, and their contribution 
to it is dependent on their weight factors which are calculated during the robust 

20 optimization. 

Referring particularly to Figure 10, a loop is entered at 164.1 where 
each of the inlier points is marked using individual weight factors. Initially, the weight 
factors of the pixels are all set to 1 so that all of the pixels are inliers. At 164.2, an 
optimization parameter, a, which determines the sensitivity of the procedure to outliers 

25 is set. The weight factors are dependent on this parameter, a. The lower the value 
of <r, the more points are eliminated as inliers and the closer the inliers become to the 
current estimate of the motion flow vector. Hence, a large a is used initially so that 
all points are included. On successive loops, a is lowered to eliminate more and more 
outliers. This lowering of a is referred to as a scheduling. The a scheduling must be 

30 done carefully. If a is lowered too fast, a solution may be missed, while on the other 
hand, lowering a too slowly increases the processing time. In accordance with the 
invention, a is lowered depending upon the largest error in the motion flow parameters. 
Following this, another loop is entered at 164.3 in which each of the inlier data points 

13 
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is used in the calculation of the updated values for the transformation parameters of the 
Q matrix at 164.4. The equations used at 164.4 are derived preferably using the 
conjugate gradient, although gradient descent can also be used. In addition, motion 
flow and robust statistics are used in deriving equations for determining the 
5 transformation parameters. The error in the transformation parameters, which is the 
change from the last calculation, as well as a, are used at 164.5 to adjust the weight 
factors for the pixels. When all of the inlier data points/pixels have been used as 
determined at 164.3, a check is made at 164.6 to determine if the solution has 
converged to the desired degree. If not, the routine returns to 164. 1 and the inlier data 

10 points are again marked using the updated weight factors. 

Figure 11 illustrates the tracking routine on 140. As indicated at 141, 
the incremental updates and the transform H and/or Q are combined so that the 
transform always relates back to the original simulation or reference image. On the 
initial pass through the tracking routine, the then current portal image replaces the 

15 simulation image if used, and then a new portal image is acquired at 143. As tracking 

continues, successive portal images are matched with the next preceding portal image 
to generate the updated transform. As indicated at 144, the successive positions of the 
fiducials or changes in the pattern of the fiducials from successive portal images is used 
to generate tracking signals for controlling the radiotherapy equipment such as turning 

20 the beam on and off and/or driving the patient positioning assembly. 

While specific embodiments of the invention have been described in 
detail, it will be appreciated by those skilled in the art that various modifications and 
alternatives to those details could be developed in light of the overall teachings of the 
disclosure. Accordingly, the particular arrangements disclosed are meant to be 

25 illustrative only and not limiting as to the scope of invention which is to be given the 
full breadth of the claims appended and any and all equivalents thereof. 



